Almost every company in the world faces the challenge of storing, retrieving and managing data in a cost-effective way with the appropriate levels of security, availability and response time. While for a variety of data, large databases are the only meaningful technology, for a significant set of data the idea of leveraging unused storage space available on clients and servers within the company seems appealing. First attempts have been made in scenarios where computer systems are connected to centrally co-ordinated computational grids, such as in the Large Hadron Collider project at CERN.
On the Internet, radically decentralized peer-to-peer architectures gained prominence through their use for file sharing applications such as the Gnutella system. Today, a large set of algorithms and programs are available for splitting, storing, and retrieving data in peer-to-peer networks, and it seems attractive to use these technologies within large companies to store and retrieve data using the available storage space on existing servers or clients in a cost-effective manner.
Among the algorithms available are Chord/DHash
(http://pdos.csail.mit.edu/papers/chord:sigcomm01/chord_sigcomm.pdf), CAN (http://www.icir.org/sylvia/thesis.ps), Pastry (http://www.cs.rice.edu/˜druschel/publications/Pastry.pdf), Tapestry (http://www.cs.berkeley.
edu/˜ravenben/publications/CSD-01-1141.pdf), and Kademlia (http://www.cs.rice.edu/ Conferences/IPTPS02/109.pdf).
A common trait of all peer-to-peer networks implemented by the mentioned algorithms is to assign to every computer system in the network a unique peer identifier from an address space, and to assign to every data object stored in the network a unique data identifier from the same address space. The address space is a metric space, a set where a notion of distance between elements of the set is defined. The topology of the address space and the way the distance is calculated vary between the available algorithms. For each data object the peer having the identifier with the smallest distance to the data object's data identifier is chosen as storage location. The peer identifiers are usually constructed as values of a hash function such as SHA-1, taking arbitrary unique properties such as the IP or MAC address of the peer computer system as input. The data identifiers of data objects are correspondingly calculated as values of the hash function, taking unique properties of the data object such as its binary contents as input.
In a peer-to-peer network realized by one of the algorithms mentioned above, stored data objects are retrieved by comparing bit-by-bit the data identifier of the data object searched for to the peer identifier of one of the peer computer systems. If the peer computer system does not possess a copy of the data object searched for but has knowledge of a further peer computer system with a peer identifier that is closer to the data identifier searched for than its own peer identifier, the query for the data object is passed on to the further peer computer system. This is repeated until the query reaches a peer computer system that possesses a copy of the data object, and the data object is retrieved.
To guard against loss of data objects in the case of failure of one or more peer computer systems, most peer-to-peer algorithms store data objects redundantly. In the simplest approach, a predetermined number of copies of each data object are stored. In some cases, so-called information dispersal algorithms are used that aim at balancing resource usage and achieved safety benefit.
Microsoft Research's Farsite
(http://research.microsoft.com/research/sn/Farsite/OSDI2002.pdf) is a secure, scalable file system that logically functions as a centralised file server but is physically distributed among a set of untrusted computers. Farsite provides file availability and reliability through randomised replicated storage; it ensures the secrecy of file contents with cryptographic techniques; it maintains the integrity of file and directory data with a Byzantine-fault-tolerant protocol; it is designed to be scalable by using a distributed hint mechanism and delegation certificates for pathname translations; and it achieves performance by locally caching file data, lazily propagating file updates, and varying the duration and granularity of content leases.
HiveCache (http://www.wtc-sf.org/IT%20Venture%20Series%20Companies.html, http://www.hivecache.com/) provides peer-to-peer distributed storage technology that aggregates the unused storage on enterprise PCs into a shared network, in order to deliver low-cost, reliable, secure storage to enterprises. HiveCache's storage mesh provides disk-based backup/restore services to desktop and laptop PCs within the enterprise, which enable users to restore their own files without needing to call in IT staff assistance and providing IT staff with a backup tool that has almost no management overhead, using policy-based decision-making by the desktop agents to maintain the storage network. These solutions use distributed resource allocation mechanisms, reputation tools, and secure distributed storage technology developed by the company and described in US patent application 20010037311.
The Distributed Internet Backup System
(http://www.csua.berkeley.edu/˜emin/source_code/dibs) enables to perform incremental back-ups of files to peers and in return store their files. It includes a peer finder service to allow finding peers to exchange backup space with. The Distributed Internet Backup System encrypts all data transmissions so that the peers files are traded with cannot access the backed-up data.